Multi-Class Image Classification Model for American Sign Language Alphabet Using TensorFlow Take 1

David Lowe

November 7, 2022

SUMMARY: This project aims to construct a predictive model using a TensorFlow convolutional neural network (CNN) and document the end-to-end steps using a template. The American Sign Language Alphabet Dataset is a multi-class classification situation where we attempt to predict one of several (more than two) possible outcomes.

INTRODUCTION: The dataset contains over 22,000 images of alphabets from American Sign Language, separated into 29 folders that represent the various classes. The research team collected these images to investigate the possibilities of reducing the communication gap between sign-language users and non-Sign language users.

ANALYSIS: The Xception model's performance achieved an accuracy score of 99.23% after three epochs using the training dataset. When we applied the model to the validation dataset, the model achieved an accuracy score of 89.94%.

CONCLUSION: In this iteration, the TensorFlow Xception CNN model appeared suitable for modeling this dataset.

Dataset ML Model: Multi-Class classification with numerical features

Dataset Used: American Sign Language Alphabet Dataset

Dataset Reference: https://www.kaggle.com/datasets/debashishsau/aslamerican-sign-language-aplhabet-dataset

One source of potential performance benchmarks: https://www.kaggle.com/datasets/debashishsau/aslamerican-sign-language-aplhabet-dataset/code

Task 1 - Prepare Environment

# Retrieve CPU information from the system
ncpu = !nproc
print("The number of available CPUs is:", ncpu[0])

The number of available CPUs is: 2

# Retrieve memory configuration information
from psutil import virtual_memory
ram_gb = virtual_memory().total / 1e9
print('Your runtime has {:.1f} gigabytes of available RAM\n'.format(ram_gb))

Your runtime has 13.6 gigabytes of available RAM

# Retrieve GPU configuration information
gpu_info = !nvidia-smi
gpu_info = '\n'.join(gpu_info)
print(gpu_info)

Fri Nov  4 17:08:30 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   69C    P0    29W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

1.a) Load libraries and modules

# Set the random seed number for reproducible results
RNG_SEED = 888

import random
random.seed(RNG_SEED)
import numpy as np
np.random.seed(RNG_SEED)
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import os
import sys
import math
# import boto3
import zipfile
from datetime import datetime
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score

import tensorflow as tf
tf.random.set_seed(RNG_SEED)
from tensorflow import keras
from tensorflow.keras.callbacks import ReduceLROnPlateau
from tensorflow.keras.preprocessing.image import ImageDataGenerator

1.b) Set up the controlling parameters and functions

# Begin the timer for the script processing
START_TIME_SCRIPT = datetime.now()

# Set up the number of CPU cores available for multi-thread processing
N_JOBS = 1

# Set up the flag to stop sending progress emails (setting to True will send status emails!)
NOTIFY_STATUS = False

# Set the percentage sizes for splitting the dataset
TEST_SET_RATIO = 0.2
VAL_SET_RATIO = 0.2

# Set the number of folds for cross validation
N_FOLDS = 5
N_ITERATIONS = 1

# Set various default modeling parameters
DEFAULT_LOSS = 'categorical_crossentropy'
DEFAULT_METRICS = ['accuracy']
DEFAULT_OPTIMIZER = tf.keras.optimizers.Adam(learning_rate=0.0001)
CLASSIFIER_ACTIVATION = 'softmax'
MAX_EPOCHS = 3
BATCH_SIZE = 16
# CLASS_LABELS = []
# CLASS_NAMES = []
# RAW_IMAGE_SIZE = (250, 250)
TARGET_IMAGE_SIZE = (299, 299)
INPUT_IMAGE_SHAPE = (TARGET_IMAGE_SIZE[0], TARGET_IMAGE_SIZE[1], 3)

# Define the labels to use for graphing the data
TRAIN_METRIC = "accuracy"
VALIDATION_METRIC = "val_accuracy"
TRAIN_LOSS = "loss"
VALIDATION_LOSS = "val_loss"

# Define the directory locations and file names
STAGING_DIR = 'staging/'
TRAIN_DIR = 'staging/ASL_Alphabet_Dataset/asl_alphabet_train'
# VALID_DIR = ''
TEST_DIR = 'staging/ASL_Alphabet_Dataset/asl_alphabet_test'
TRAIN_DATASET = 'archive.zip'
# VALID_DATASET = ''
# TEST_DATASET = ''
# TRAIN_LABELS = ''
# VALID_LABELS = ''
# TEST_LABELS = ''
# OUTPUT_DIR = 'staging/'
# SAMPLE_SUBMISSION_CSV = 'sample_submission.csv'
# FINAL_SUBMISSION_CSV = 'submission.csv'

# Check the number of GPUs accessible through TensorFlow
print('Num GPUs Available:', len(tf.config.list_physical_devices('GPU')))

# Print out the TensorFlow version for confirmation
print('TensorFlow version:', tf.__version__)

Num GPUs Available: 1
TensorFlow version: 2.9.2

# Set up the email notification function
def status_notify(msg_text):
    access_key = os.environ.get('SNS_ACCESS_KEY')
    secret_key = os.environ.get('SNS_SECRET_KEY')
    aws_region = os.environ.get('SNS_AWS_REGION')
    topic_arn = os.environ.get('SNS_TOPIC_ARN')
    if (access_key is None) or (secret_key is None) or (aws_region is None):
        sys.exit("Incomplete notification setup info. Script Processing Aborted!!!")
    sns = boto3.client('sns', aws_access_key_id=access_key, aws_secret_access_key=secret_key, region_name=aws_region)
    response = sns.publish(TopicArn=topic_arn, Message=msg_text)
    if response['ResponseMetadata']['HTTPStatusCode'] != 200 :
        print('Status notification not OK with HTTP status code:', response['ResponseMetadata']['HTTPStatusCode'])

if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 1 - Prepare Environment completed on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))

Task 2 - Load and Prepare Images

if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 2 - Load and Prepare Images has begun on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))

# Clean up the old files and download directories before receiving new ones
!rm -rf staging/

if not os.path.exists(TRAIN_DATASET):
    !wget https://dainesanalytics.com/datasets/kaggle-debashishsau-sign-language-aplhabet/archive.zip

zip_ref = zipfile.ZipFile(TRAIN_DATASET, 'r')
zip_ref.extractall(STAGING_DIR)
zip_ref.close()

CLASS_LABELS = os.listdir(TRAIN_DIR)
print(CLASS_LABELS)
NUM_CLASSES = len(CLASS_LABELS)
print('Total number of classes detected:', NUM_CLASSES)

['O', 'F', 'Y', 'X', 'V', 'I', 'P', 'space', 'nothing', 'J', 'R', 'A', 'K', 'B', 'M', 'T', 'D', 'Q', 'N', 'E', 'del', 'L', 'H', 'C', 'G', 'U', 'S', 'Z', 'W']
Total number of classes detected: 29

# Brief listing of training image files for each class
for c_label in CLASS_LABELS:
    training_class_dir = os.path.join(TRAIN_DIR, c_label)
    training_class_files = os.listdir(training_class_dir)
    print('Number of training images for', c_label, ':', len(os.listdir(training_class_dir)))
    print('Training samples for', c_label, ':', training_class_files[:5],'\n')

Number of training images for O : 8140
Training samples for O : ['O (29).jpg', 'O (3830).jpg', 'O1036.jpg', 'O2505.jpg', 'O1629.jpg'] 

Number of training images for F : 8031
Training samples for F : ['F (1504).jpg', 'F937.jpg', 'F (3480).jpg', 'F1587.jpg', 'F (2422).jpg'] 

Number of training images for Y : 8178
Training samples for Y : ['Y (3607).jpg', 'Y2686.jpg', 'y_41_rotate_2.jpeg', 'Y944.jpg', 'Y (2911).jpg'] 

Number of training images for X : 8093
Training samples for X : ['X414.jpg', 'X454.jpg', 'X (2250).jpg', 'X (1825).jpg', 'X (3878).jpg'] 

Number of training images for V : 7597
Training samples for V : ['V (1473).jpg', 'V1471.jpg', 'V (1164).jpg', 'V (1160).jpg', 'V1197.jpg'] 

Number of training images for I : 7953
Training samples for I : ['I (1047).jpg', 'I1098.jpg', 'i_67_rotate_4.jpeg', 'I2583.jpg', 'I (2367).jpg'] 

Number of training images for P : 7601
Training samples for P : ['P (1331).jpg', 'P2084.jpg', 'P (62).jpg', 'P (2728).jpg', 'P1842.jpg'] 

Number of training images for space : 7071
Training samples for space : ['space1973.jpg', 'space (3307).jpg', 'space1479.jpg', 'space (1085).jpg', 'space (3057).jpg'] 

Number of training images for nothing : 3030
Training samples for nothing : ['nothing1763.jpg', 'nothing1086.jpg', 'nothing1391.jpg', 'nothing783.jpg', 'nothing471.jpg'] 

Number of training images for J : 7503
Training samples for J : ['j_27_rotate_7.jpeg', 'J1931.jpg', 'J (1316).jpg', 'J2703.jpg', 'J1564.jpg'] 

Number of training images for R : 8021
Training samples for R : ['R (3978).jpg', 'R (613).jpg', 'R (1870).jpg', 'R2176.jpg', 'R2411.jpg'] 

Number of training images for A : 8458
Training samples for A : ['A96.jpg', 'A (3334).jpg', 'A30.jpg', 'A (3943).jpg', 'A (1809).jpg'] 

Number of training images for K : 7876
Training samples for K : ['K (72).jpg', 'K2571.jpg', 'k_10_rotate_3.jpeg', 'K1516.jpg', 'K2386.jpg'] 

Number of training images for B : 8309
Training samples for B : ['B (1499).jpg', 'B125.jpg', 'B (429).jpg', 'B (2945).jpg', 'B (3436).jpg'] 

Number of training images for M : 7900
Training samples for M : ['M (882).jpg', 'M2458.jpg', 'm_59_rotate_8.jpeg', 'M (3435).jpg', 'M254.jpg'] 

Number of training images for T : 8054
Training samples for T : ['T346.jpg', 'T2210.jpg', 'T494.jpg', 'T (874).jpg', 'T1384.jpg'] 

Number of training images for D : 7629
Training samples for D : ['D603.jpg', 'D (310).jpg', 'D (359).jpg', 'D (2548).jpg', 'D (784).jpg'] 

Number of training images for Q : 7954
Training samples for Q : ['Q1281.jpg', 'q_61_rotate_3.jpeg', 'q_38_rotate_4.jpeg', 'Q (3941).jpg', 'Q1985.jpg'] 

Number of training images for N : 7932
Training samples for N : ['N2249.jpg', 'n_44_rotate_3.jpeg', 'N0012_test.jpg', 'N (919).jpg', 'N (149).jpg'] 

Number of training images for E : 7744
Training samples for E : ['E828.jpg', 'E749.jpg', 'E (756).jpg', 'E (1054).jpg', 'E0022_test.jpg'] 

Number of training images for del : 6836
Training samples for del : ['del880.jpg', 'del796.jpg', 'del (602).jpg', 'del (1448).jpg', 'del (544).jpg'] 

Number of training images for L : 7939
Training samples for L : ['L2890.jpg', 'L (2095).jpg', 'L (3164).jpg', 'L (3614).jpg', 'L1463.jpg'] 

Number of training images for H : 7906
Training samples for H : ['H (967).jpg', 'H2558.jpg', 'h_31_rotate_3.jpeg', 'h_13_rotate_2.jpeg', 'H1609.jpg'] 

Number of training images for C : 8146
Training samples for C : ['C (1707).jpg', 'C1004.jpg', 'c_60_rotate_10.jpeg', 'c_15_rotate_6.jpeg', 'C (3150).jpg'] 

Number of training images for G : 7844
Training samples for G : ['G (2586).jpg', 'G (2349).jpg', 'G (798).jpg', 'G2503.jpg', 'G (3509).jpg'] 

Number of training images for U : 8023
Training samples for U : ['U (3741).jpg', 'U (1709).jpg', 'U269.jpg', 'U (1264).jpg', 'U (355).jpg'] 

Number of training images for S : 8109
Training samples for S : ['S2762.jpg', 'S563.jpg', 'S2659.jpg', 'S (2816).jpg', 'S1002.jpg'] 

Number of training images for Z : 7410
Training samples for Z : ['Z (694).jpg', 'Z753.jpg', 'Z (828).jpg', 'Z1739.jpg', 'Z655.jpg'] 

Number of training images for W : 7787
Training samples for W : ['W (960).jpg', 'W (2147).jpg', 'W1504.jpg', 'W2400.jpg', 'W1614.jpg'] 

# Plot some training images from the dataset
nrows = len(CLASS_LABELS)
ncols = 4
training_examples = []
example_labels = []

fig = plt.gcf()
fig.set_size_inches(ncols * 4, nrows * 3)

for c_label in CLASS_LABELS:
    training_class_dir = os.path.join(TRAIN_DIR, c_label)
    training_class_files = os.listdir(training_class_dir)
    for j in range(ncols):
        training_examples.append(training_class_dir + '/' + training_class_files[j])
        example_labels.append(c_label)
    # print(training_examples)
    # print(example_labels)

for i, img_path in enumerate(training_examples):
    # Set up subplot; subplot indices start at 1
    sp = plt.subplot(nrows, ncols, i+1)
    sp.text(0, 0, example_labels[i])
    # sp.axis('Off')
    img = mpimg.imread(img_path)
    plt.imshow(img)
plt.show()

datagen_kwargs = dict(rescale=1./255, validation_split=VAL_SET_RATIO)
training_datagen = ImageDataGenerator(**datagen_kwargs)
validation_datagen = ImageDataGenerator(**datagen_kwargs)
dataflow_kwargs = dict(class_mode="categorical")

do_data_augmentation = True
if do_data_augmentation:
    training_datagen = ImageDataGenerator(rotation_range=45,
                                          horizontal_flip=True,
                                          vertical_flip=True,
                                          **datagen_kwargs)

print('Loading and pre-processing the training images...')
training_generator = training_datagen.flow_from_directory(directory=TRAIN_DIR,
                                                          target_size=TARGET_IMAGE_SIZE,
                                                          batch_size=BATCH_SIZE,
                                                          shuffle=True,
                                                          seed=RNG_SEED,
                                                          subset="training",
                                                          **dataflow_kwargs)
print('Number of training image batches per epoch of modeling:', len(training_generator))

print('Loading and pre-processing the validation images...')
validation_generator = validation_datagen.flow_from_directory(directory=TRAIN_DIR,
                                                              target_size=TARGET_IMAGE_SIZE,
                                                              batch_size=BATCH_SIZE,
                                                              shuffle=False,
                                                              subset="validation",
                                                              **dataflow_kwargs)
print('Number of validation image batches per epoch of modeling:', len(validation_generator))

Loading and pre-processing the training images...
Found 178472 images belonging to 29 classes.
Number of training image batches per epoch of modeling: 11155
Loading and pre-processing the validation images...
Found 44602 images belonging to 29 classes.
Number of validation image batches per epoch of modeling: 2788

if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 2 - Load and Prepare Images completed on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))

Task 3 - Define and Train Models

if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 3 - Define and Train Models has begun on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))

# Define the function for plotting training results for comparison
def plot_metrics(history):
    fig, axs = plt.subplots(1, 2, figsize=(24, 15))
    metrics =  [TRAIN_LOSS, TRAIN_METRIC]
    for n, metric in enumerate(metrics):
        name = metric.replace("_"," ").capitalize()
        plt.subplot(2,2,n+1)
        plt.plot(history.epoch, history.history[metric], color='blue', label='Train')
        plt.plot(history.epoch, history.history['val_'+metric], color='red', linestyle="--", label='Val')
        plt.xlabel('Epoch')
        plt.ylabel(name)
        if metric == TRAIN_LOSS:
            plt.ylim([0, plt.ylim()[1]])
        else:
            plt.ylim([0, 1])
        plt.legend()

# Define the baseline model for benchmarking
def create_nn_model(input_param=INPUT_IMAGE_SHAPE, output_param=NUM_CLASSES, dense_nodes=2048,
                    classifier_activation=CLASSIFIER_ACTIVATION, loss_param=DEFAULT_LOSS,
                    opt_param=DEFAULT_OPTIMIZER, metrics_param=DEFAULT_METRICS):
    base_model = keras.applications.xception.Xception(include_top=False, weights='imagenet', input_shape=input_param)
    nn_model = keras.models.Sequential()
    nn_model.add(base_model)
    nn_model.add(keras.layers.Flatten())
    nn_model.add(keras.layers.Dense(dense_nodes, activation='relu')),
    nn_model.add(keras.layers.Dense(output_param, activation=classifier_activation))
    nn_model.compile(loss=loss_param, optimizer=opt_param, metrics=metrics_param)
    return nn_model

# Initialize the neural network model and get the training results for plotting graph
start_time_module = datetime.now()
tf.keras.utils.set_random_seed(RNG_SEED)
baseline_model = create_nn_model()
baseline_model_history = baseline_model.fit(training_generator,
                                            epochs=MAX_EPOCHS,
                                            validation_data=validation_generator,
                                            verbose=1)
print('Total time for model fitting:', (datetime.now() - start_time_module))

Epoch 1/3
11155/11155 [==============================] - 7579s 678ms/step - loss: 0.2965 - accuracy: 0.9152 - val_loss: 0.4181 - val_accuracy: 0.9009
Epoch 2/3
11155/11155 [==============================] - 7564s 678ms/step - loss: 0.0417 - accuracy: 0.9894 - val_loss: 0.5608 - val_accuracy: 0.8894
Epoch 3/3
11155/11155 [==============================] - 7541s 676ms/step - loss: 0.0306 - accuracy: 0.9923 - val_loss: 0.5025 - val_accuracy: 0.8994
Total time for model fitting: 6:18:16.928629

baseline_model.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 xception (Functional)       (None, 10, 10, 2048)      20861480  
                                                                 
 flatten (Flatten)           (None, 204800)            0         
                                                                 
 dense (Dense)               (None, 2048)              419432448 
                                                                 
 dense_1 (Dense)             (None, 29)                59421     
                                                                 
=================================================================
Total params: 440,353,349
Trainable params: 440,298,821
Non-trainable params: 54,528
_________________________________________________________________

plot_metrics(baseline_model_history)

# if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 3 - Define and Train Models completed on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))

Task 4 - Tune and Optimize Models

# if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 4 - Tune and Optimize Models has begun on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))

# Not applicable for this iteration of modeling

# if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 4 - Tune and Optimize Models completed on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))

Task 5 - Finalize Model and Make Predictions

# if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 5 - Finalize Model and Make Predictions has begun on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))

# Not applicable for this iteration of modeling

# if NOTIFY_STATUS: status_notify('(TensorFlow Multi-Class) Task 5 - Finalize Model and Make Predictions completed on ' + datetime.now().strftime('%A %B %d, %Y %I:%M:%S %p'))

print ('Total time for the script:',(datetime.now() - START_TIME_SCRIPT))

Total time for the script: 6:19:53.276701